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Abstract 

This study compared learning for fifth grade students in two math homework conditions. 

The paper-and-pencil condition represented traditional homework, with review of prob- 
lems in class the following day. The Web-based homework condition provided immediate 
feedback in the form of hints on demand and step-by-step scaffolding. We analyzed the re- 
sults for students who completed both the paper-and-pencil and the Web-based conditions. 

In this group of 28 students, students learned significantly more when given computer 
feedback than when doing traditional paper-and-pencil homework, with an effect size of 
.61. The implications of this study are that, given the large effect size, it may be worth 
the cost and effort to give Web-based homework when students have access to the need- 
ed equipment, such as in schools that have implemented one-to-one computing programs. 
(Keywords: online homework, intelligent tutoring systems, online tutoring, homework.) 

Web-based homework assistance is already popular in colleges. Blackboard 
(www.blackboard.com), WebAssign, (www.webassign.com), MasteringPhys- 
ics (www.masteringphysics.com), and WebWorK (http://webwork.rochester. 
edu) are all systems that have thousands of student users at the college level, 
but K— 12 Web-based homework assistance lags behind. Systems such as Study 
Island (www.studyisland.com) and PowerSchool (www.powerschool.com) are 
gaining popularity with K-12 teachers, and it seems likely that the use of Web- 
based homework assistance for K-12 will increase as the digital divide between 
students narrows, teachers become more comfortable with the technology, and 
teachers gain access to systems that are low cost or free. The important question 
is, do such systems help students learn more than traditional paper-and-pencil 
homework? 

With recent advances in educational technology, teachers now have a multi- 
tude of tools to assist and enhance student learning and motivation. New intel- 
ligent tutoring systems that guide students through math problems much the 
same way human tutors do have been successful in helping students learn math 
in the classroom. Some systems attempt to imitate a human tutor by reproduc- 
ing the interactive dialogue patterns and strategies that were likely to be used 
by a human tutor, whereas others provide immediate feedback by highlighting 
each step attempted in either red or green to indicate a right or wrong answer. 
They may also provide hint sequences to students asking for help. 
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A U.S. Congress-mandated study (Dynarski et al., 2007) reported that 
classrooms using selected math and reading educational software products did 
not differ significantly on standardized tests when compared to classrooms that 
did not use the products. This study has caused some to call into question the 
utility of educational software, which might suggest that we should stop wasting 
time producing computer-based systems. However, the Dynarski study looked 
at computers used during the school day and not as a homework-delivery 
system. 

In this study, we attempt to determine if fifth grade students can learn more 
by doing their math homework with a Web-based intelligent tutoring system 
than when doing traditional paper-and-pencil homework. We conducted this 
experiment using the ASSISTment System, a Web-based system that provides 
both interactive scaffolding and hints on demand. We will review studies of 
Web-based homework assistance systems before describing the ASSISTment 
system used in this evaluation. 

LITERATURE REVIEW 

The Use of Web-based Systems for Homework 

Web-based systems that allow students to do their homework online such as 
Blackboard, WebCT (www.webct.com), Homework Service (https://hw.utexas. 
edu/bur/overview.html), and WebWorK are becoming more widely used in 
higher education. At the K-12 level, systems such as Study Island and Power- 
School are gaining popularity among teachers. 

Some states in the United States, including Maine, Indiana, Michigan, and 
Virginia, have begun to implement one-to-one computing (Bonifaz & Zucker, 
2004) in schools where each child gets his/her own laptop to use during school 
hours and often to take home. For instance, the Maine Learning Technol- 
ogy Initiative (2002-2004) supplied every seventh and eighth grade student 
in Maine and their teachers with laptop computers, and 40% of the middle 
schools allow students to take their laptops home. Although few research stud- 
ies on the effects of one-to-one computing on teaching and learning have been 
reported, teachers report that students in one-to-one computing programs are 
more engaged and motivated and interact better with teachers (Bebell, 2005; 
Silvernail, & Lane, 2004). At the same time, recommendations for abandon- 
ing one-to-one computing programs citing the high cost, potential access to 
inappropriate material, and lack of proven impact on student achievement (Hu, 
2007; Vascellaro, 2006) have been widely published. Still, the number of U.S. 
schools adopting one-to-one computing programs continues to increase every 
year, according to a survey of the largest 2,500 school districts in the United 
States conducted by the Hayes Connection and cited in the New York Times 
by Hu (2007). The opportunities for students to do their homework online 
increase as the digital divide narrows and more states become committed to 
one-to-one computing. One important question is, do students learn more by 
using computers to do their homework than by doing traditional paper-and- 
pencil homework? 
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Some advantages of homework-assistance systems are immediate feedback 
to students and automatic grading and recording of grades for instructors. 
Automatic grading saves time for teachers who would like to grade all of their 
students’ paper-and-pencil homework carefully by hand but do not have time. 
In turn, this can prompt students to take homework more seriously because 
they know it will be graded and the grade will be recorded. With these systems, 
students can often get immediate feedback on their answers and work and 
sometimes help toward solving problems. 

Although these Web-based homework-assistance systems can provide ben- 
efits, they can have disadvantages, as well. Many of these systems do not take 
students’ work into consideration when they require students to enter a single 
answer for each problem. Students may be less organized because they do less 
scrap work on paper and try to do more math in their heads. Teachers may 
be less able to figure out exactly where students are having difficulties without 
seeing their work. Finally, because these systems often do not consider student 
work, cheating may be easier among students because they could possibly get 
the answers from their friends without having to show how they arrived at 
them. 

Web-based Homework Versus Paper-and- Pencil Homework 

Previous research has shown positive results for using Web-based homework 
assistance instead of traditional paper-and-pencil homework. MasteringPhysics, 
a Web-based physics homework tutor developed at MIT, uses mastery learning 
to help students reach mastery when solving physics homework problems. Stu- 
dents can ask for hints on problems and receive feedback on common student 
errors. Some hints will ask the student a question that behaves like a “scaf- 
folding question” in the ASSISTment system (described in the next section). 
Warnakulasooriya & Pritchard (2005) found that twice as many students could 
complete a set of problems in a given time with the help provided by Master- 
ingPhysics when compared to students who worked on the problems without 
help (administered by MasteringPhysics but without hints or feedback). 

The Andes system is an intelligent tutoring system that provides support for 
problem solving for physics homework. Students using Andes complete whole 
derivations step by step and receive feedback after each step. Students can also 
request hints for each step to find out where their errors are (What’s Wrong 
Help) or to find out what to do next (Next Step Help). VanLehn et al. (2005) 
presented evidence from introductory physics courses taught at the U.S. Naval 
Academy from 1999 to 2003 that students who used Andes for homework got 
significantly higher exam scores than students in control groups who did paper- 
and-pencil homework. Other studies of Web-based physics homework versus 
paper-and-pencil homework did not find significant differences between the 
two homework conditions (Bonham, Deardorff, & Beichner, 2003; Pascarella, 
2002). The Andes studies seem to be the most closely related to our comparison 
of ASSISTments and paper-and-pencil homework, and this study attempts to 
replicate Andes’ positive results. 


Journal of Research on Technology in Education 

Copyright © 2009, ISTE (International Society for Technology in Education), 800.336.5191 
(U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. 


333 



The ASSISTment System 

Assistance and assessment are integrated in a Web-based system called the 
ASSISTment System, which offers instruction to students while providing a 
detailed evaluation of their abilities to teachers. Each time students work on the 
Web site, the system “learns” more about their abilities. The ASSISTment Sys- 
tem is being built to identify the difficulties individual students — and the class 
as a whole — are having, and teachers will be able to use this detailed feedback 
to tailor their instruction to focus on those difficulties. Unlike other assessment 
systems, the ASSISTment system also provides students with intelligent tutoring 
assistance while assessment information is collected. Our hypothesis is that AS- 
SISTments can do a better job of getting students to learn from homework than 
traditional paper-and-pencil homework by providing immediate feedback on 
answers and tutoring for items that students get wrong. Teachers can also assess 
their students’ knowledge limitations better by noting the amount and nature 
of the assistance students need to finish their homework. The ASSISTment 
system was developed using grants from the U.S. Department of Education and 
the National Science Foundation and will be freely available for use by teachers 
and students. 

The ASSISTment System provides students with two types of tutoring assis- 
tance, scaffolds and hints, when they answer a question incorrectly. With scaf- 
folds, the student who answers an ASSISTment incorrectly receives the message, 
“Hmm, no. Let me break this down for you,” and is immediately presented 
with a scaffolding question that the student must answer correctly in order to 
continue and receive the next scaffolding question. The scaffolding questions 
break the problem into steps, and students must answer the scaffolding ques- 
tions before returning to the original question; that is, the student is forced 
to work through the problem. With hints, the student may request a hint by 
pressing the hint button. Students may request more hints until they reach the 
“bottom-out” hint, which will typically give them the answer to the question. 

When students log in to the ASSISTment system, they are presented with 
math problems. Figure 1 shows a screenshot of an ASSISTment problem with 
three scaffolding questions. Solving this problem involved understanding con- 
gruence, perimeter, and equation solving. If the student had answered correctly, 
she would have moved on to a new problem. FFowever, she incorrectly answered 
23, and the system responded with, “FFmrn, no. Let me break this down for 
you.” It then presented the student with some questions that would help to 
isolate the skills with which she had difficulty and to tutor her so that she could 
figure out the correct actions. The tutor began by asking a scaffolding question 
that isolated the step involving congruence. Eventually she got the scaffolding 
question correct (by answering “AC”) and then was given a question about pe- 
rimeter. The figure shows that the student selected Vi * 8 * x as the formula for 
perimeter, and the system responded with a “buggy message” letting the student 
know she seems to be confusing perimeter with area. The student requested two 
hint messages, as shown at the bottom of the screen. The tutoring ends with a 
final question, which is actually the original question asked again. The student 
then will go on to do another math problem and will again get tutoring if she 
gets it wrong. 
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File Edit View Favorites Tools Help 


; Q Back 0 0 *£*) ^Search "^Favortes 0 - _ ; Links 

I Address Jt} http://www.assistment.org/portal/index.jsp V 0G« 


inches 
2x C 



Triangles ABC and DEF are congruent. 
The perimeter of triangle ABC is 
23 inches. What is the length of 
side DF in triangle DEF? 

Hmm, no. 

Let me break this down for you. 


The original question 

a. Congruence 

b. Perimeter 

c. Equation-Solving 


Which side of triangle ABC has the same length i 
as side DF of triangle DEF? 


The 1 st scaffolding question 


Congruence 


What is the perimeter of triangle ABC? 
02x + 8 
©‘/ 2 *8x 
O 2x + x + 8 
O ’/ 2 * x(2x) 


A buggy message 


The 2 nd scaffolding question 

I Perimeter I 


No. You might be thinking that the area is 1/2 base times height, but you are looking for the 
perimeter. 

Perimeter is defined as the sum of all sides of a figure. 

a The 1 st hint message 

The perimeter of tnangle ABC is the sum of all its sides. 

lv 


"■ The 2 nd hint message 

Done 


Figure 1. An ASSISTment showing three scaffolding questions. 
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The design of the ASSISTment system was informed by earlier systems such 
as Cognitive Tutors (Anderson et ah, 1995) and Ms. Lindquist (Heffernan & 
Koedinger, 2002). These systems use “model-tracing” architectures that were 
invented by researchers at Carnegie Mellon University (Anderson, Boyle & 
Reiser, 1985; Anderson & Pelletier, 1991) and used extensively to build tutors. 
Each tutor is constructed around a cognitive model of the problem-solving 
knowledge students have and the knowledge needed to solve each problem. The 
model reflects the ACT-R theory of skill knowledge (Anderson, 1993), which 
assumes that problem-solving skills can be modeled as a set of independent pro- 
duction rules. Production rules are if-then rules that represent different pieces 
of knowledge. 

Ms. Lindquist, developed by Heffernan and Koedinger (2002), is an intelli- 
gent tutoring system that uses dialog to help students write algebra expressions. 
Ms. Lindquist models both student behavior and tutorial behavior by combin- 
ing a cognitive model of student behavior with a tutorial model of strategies 
observed in a human tutor. The cognitive student model has a set of production 
rules that model the problem-solving skills needed to write algebraic expres- 
sions, whereas the tutorial model is based on the observation of an experienced 
human tutor during a tutoring session and tries to capture tutorial strategies 
that were observed to be effective. Ms. Lindquist was the first intelligent tutor 
that had both a model of student thinking and a model of tutorial planning 
and is different from typical Cognitive Tutors in that it takes its cues more from 
the dialogues that human tutors have with students and is more flexible. For 
example, it can acknowledge that part of an answer is correct and then engage 
a student in a “sub-dialogue” to help him or her improve the incorrect path. It 
“breaks” problems down for students by asking questions and rephrasing ques- 
tions but does not give students answers. 

The ASSISTment system also breaks problems down for students in the way 
that Cognitive Tutors and Ms. Lindquist do, but it is not rule based. Koedinger 
et al. (2004) introduced example-tracing tutors that mimic Cognitive Tutors 
but are limited to the scope of a single problem. The ASSISTment system uses 
a further simplified example-tracing tutor called an ASSISTment that allows 
only a linear progression through a problem, which makes content creation 
easier and more accessible to nonprogrammers. Previous results show that our 
example-tracing-based system can reduce the time required to build a single 
hour of content from 100-1,000 hours to 10-30 hours (Razzaq et al., 2008). 

During the design stage of the ASSISTment system, middle school math 
teachers were involved in the process (Razzaq et al., 2005). Heffernan and 
Razzaq met with these teachers weekly to do “knowledge elicitation” inter- 
views, during which the teachers helped design the pedagogical content of the 
ASSISTment system. The knowledge elicitation interviews started by showing 
the teacher a problem and then asking a series of questions: “How would you 
tutor a student to solve the problem? How would you break the problem down? 
What hints would you give the student? What kinds of errors are expected, and 
what would you say when a student made an expected error?” They videotaped 
these interviews and used the recordings to fill out an “ASSISTment design 
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Table 1. Participants in the Study 



Class A 

Class B 

Class C 

Class D 

Total 

Male students 

13 

12 

9 

12 

46 

Female students 

10 

13 

13 

10 

46 

Students with Internet at home 

12 

15 

17 

10 

54 


form,” which they used to implement the ASSISTment. Teachers gave their 
opinions on the first draft of the ASSISTment and were asked to edit it. They 
also videotaped these review sessions and revised the design form as needed. 
When the teacher was satisfied, they released the ASSISTment for use. 

Initial studies of the ASSISTment system (Razzaq et ah, 2005) found that 
students were learning eighth grade math while using the system once every 
two weeks during their regular math classes. The purpose of this study was to 
determine if using the ASSISTment system for Web-based homework assistance 
is more useful (produces more learning) than traditional paper-and-pencil 
homework. 

METHOD 

Setting and Participants 

The setting for this study was 4 fifth grade classrooms and students’ home 
computers. The school was located in a small town in a rural county and was a 
sample of convenience. Approximately 350 students were enrolled in the school 
at the time of the study, with at least 50% receiving free or reduced lunch. All 
four classes were typical elementary classes with a mix of below-average, aver- 
age, and above-average students. Teachers gave a total of 92 students (54 with 
Internet access at home) this homework assignment, depending on their access. 
The breakdown of the participants is shown in Table 1 . 

Content 

We used two problem sets in both the Web-based homework and the paper- 
and-pencil homework assignments, each consisting of 10 problems. One prob- 
lem set consisted of Number Sense problems, and the other was a mix of prob- 
lems. The Number Sense problem set included problems for which students 
had to demonstrate understanding of numbers, ways of representing numbers, 
and relationships among numbers and number systems. The Mixed problem 
set included problems in the algebra, geometry, data analysis, and probability 
domains. Students had to demonstrate understanding of patterns, relations, and 
functions; describe spatial relationships using coordinate geometry and other 
representational systems; develop and evaluate inferences and predictions that 
are based on models; and apply and demonstrate an understanding of basic 
concepts of probability. (See Appendix B for the problems in both homework 
sets.) Students in the classes had prior learning experience with the homework 
material during the course of the school year. However, as the experiment took 
place at the end of the school year, it was not recent experience and was more of 
a review. 
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The worksheets that students completed for paper-and-pencil homework 
assignments were identical to the Web-based homework assignment problems, 
with the same formats (i.e., multiple choice or short answer). This was possible 
because each class did the Number Sense or Mixed problem set for computer 
homework and the opposite problem set for paper-and-pencil homework. 

This will be explained in detail in the Experimental Design section below. The 
pretest and posttest items were “morphs” of the Number Sense and Mixed 
problem sets and were designed to have different surface features, such as names 
and numbers, while keeping the same deep features or knowledge requirements, 
such as analyzing patterns. Finally, the same hints used in the problem sets on 
the Web-based tutor were used while going over paper-and-pencil homework 
problems in class to ensure that each class had the same instruction. 

Experimental Design 

We used a counterbalanced experimental design in which students in two 
classrooms participated in the Web-based homework condition first, whereas 
students in the other two classrooms participated in the paper-and-pencil con- 
dition first. All students participated in both Web-based and paper-and-pencil 
conditions, and all students received pretests and posttests for each condition in 
which they participated. 

In the Web-based first group, one class received a pretest for the Number 
Sense problem set, and the other class received a pretest for the Mixed problem 
set. Students then received a homework assignment consisting of 10 problems 
in their respective problem sets on the Web-based system. After completing the 
Web-based homework, the students received posttests in class the next day. We 
then reversed the groups, with the Number Sense group receiving the Mixed 
pretest and the Mixed group receiving the Number Sense pretest. Both groups 
then received a paper-and-pencil homework assignment that consisted of 10 
problems on a worksheet for their respective problem sets. They completed 
posttests the following day. The paper-and-pencil first group participated in 
the same overall experimental design. (See Table 2 for the overall experimental 
design.) 

Our design counterbalances the content (number sense vs. mixed) as well as 
the order of condition (Web-based vs. paper and pencil) so that we can draw 
valid inferences that any gains students make will not be attributed to these out- 
side factors. 

Procedures 

On day one of the experiment, students in all four classes received the ap- 
propriate pretest (two classes were administered the Number Sense pretest, and 
two were administered the Mixed pretest). (See Appendix A for sample pre- and 
posttests.) Thus, for both conditions, Web-based homework and paper-and- 
pencil homework, one class was completing the Number Sense assignment 
while the other class completed the Mixed assignment. 

After completing the pretest, each class in the paper-and-pencil homework 
condition received worksheets with 10 problems to complete for homework. 
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Table 2. Experimental Design 


Day 

Web-based first group 


Paper-and-Pencil First Group 


Class A 

Class B 

Class C 

Class D 

Mon. 

Pretest Number 
Sense 

Pretest Mixed 
Intro to 

Pretest Mixed 
Paper-and-pencil 

Pretest Number 
Sense 


Intro to 
ASSISTments 

Web-based 

assignment 

ASSISTments 

Web-based 

assignment 

assignment 

Paper-and-pencil 

assignment 

Tues. 

Posttest Number 
Sense 

Posttest Mixed 

Review assignment 
Posttest Mixed 

Review 

assignment 

Posttest Number 
Sense 

Wed. 

Pretest Mixed 
Paper-and-pencil 

Pretest 

Number Sense 

Pretest Number 
Sense 

Pretest Mixed 
Intro to 


assignment 

Paper-and- 

pencil 

assignment 

Intro to 
ASSISTments 

Web-based 

assignment 

ASSISTments 

Web-based 

assignment 

Thurs. 

Review assignment 
Posttest Mixed 

Review 

assignment 

Posttest 
Number Sense 

Posttest Number 
Sense 

Posttest Mixed 


(See Appendix B for sample worksheet problems.) They instructed the stu- 
dents to bring the worksheets to school the following day so that they could 
go over them and answer any questions they had. Students in each class in the 
Web-based homework condition completed their pretests and then were taken 
to the media room, where they learned how to log in to and use the ASSIST- 
ment system. They all received a school identifier, which was the same for all 
students, then received an individual screen name consisting of their first name 
and last initial. After each student logged in to the system, they were shown 
how to select their teachers’ names and how to select their homework for the 
evening. They also learned how to select and work on a demonstration problem 
to familiarize themselves with the system and how problems were presented. 
The demonstration problems were not a part of their homework problems. (See 
Appendix C for sample computer/ASSISTment problems.) 

On day 2, the students in the paper-and-pencil condition received the 
answers to their worksheet problems and then had the opportunity to ask 
questions for review. (We are aware that our homework review procedure is 
susceptible to the fact that some students are more assertive than others when 


Journal of Research on Technology in Education 

Copyright © 2009, ISTE (International Society for Technology in Education), 800.336.5191 
(U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. 


339 



Table 3. Group and Individual Class Completion Rates 



Class 

A 

Class 

B 

Class 

C 

Class 

D 

All 

Classes 

Exclude 
Class D 

% with Internet 

17/24 

12/23 

10/22 

15/24 

54/93 

39/69 


70.8% 

52% 

45% 

62.5% 

58% 

56.5% 

% with Internet that 

13/17 

6/12 

7/10 

3/15 

29/54 

26/39 

completed WBH 

76% 

50% 

70% 

20% 

53.7% 

67% 

% with Internet that 

15/17 

9/12 

7/10 

12/15 

43/54 

31/39 

completed PPH 

88% 

75% 

70% 

80% 

79.6% 

79% 

% with Internet that 

12/17 

6/12 

7/10 

3/15 

28/54 

25/39 

completed both 

70% 

50% 

70% 

20% 

52% 

64% 

asking questions, but 

we argue 

that our 

design is reasonable, as we have tried 


to replicate what happens in typical classrooms when going over homework.) 
When answering questions the teacher used the exact hints used in the com- 
puter-hints condition to ensure uniformity across groups. (See Appendix D for 
sample hints.) The review was limited to 10 minutes per group. 

We designed the procedures used for the homework review to simulate a typi- 
cal review that math teachers would use. For example, in a typical math class 
the teacher will usually go over the answers for the homework assignment and 
then ask students if they have any questions. If they do, the teacher will put the 
problems on the board for students to see or from time to time have students 
do problems on the board so the teacher can see where the students are having 
difficulty. The reviews usually last approximately 10 minutes. For this study we 
attempted to simulate those procedures as closely as possible, but with a few ex- 
ceptions. For example, we went over the answers to the homework assignment, 
and then asked students if they had any questions. However, in lieu of doing 
the problems on the board, we chose to use overheads with the exact hints used 
in the ASSISTment problems. We did this because we wanted to ensure that 
each group received the same review. 

The students then completed posttests. The two classes in the computer 
condition received posttests on day 2. Days 3 and 4 followed the same proce- 
dures as days 1 and 2, but the groups were reversed. That is, the two classes in 
the Web-based condition switched with the two classes in the paper-and-pencil 
condition, and the two classes that did number-sense problems switched with 
the two classes doing mixed problems. 

Results 

There were a total of 93 students in the four classes. Of the 93 students, 5 
were absent during all or part of the study, another 6 missed part of the study 
due to activities in which they were involved outside of the classroom, and 6 
students were nonreaders. These students did not participate in the study, which 
left a total of 76 students. Only 54 of those students, however, had Internet 
available at home and could participate fully in the study. Of these 54 students, 
31 students (57%) started the Web-based homework, but 2 of them reported 
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Paired Samples Statistics 




Mean 

N 

Std. 

Deviation 

Std. 

Error 

Mean 

Pair 

Gain 1 Compter 

2.32 

28 

2.195 

415 

1 

Gain2PPH 

1.14 

28 

1.533 

290 


Paired Samples Correlations 



N 

Correlation 

Sig. 

Pair Gain 1 Compter 

1 & Gain2PPH 

28 

-.322 

.094 


Paired Samples Test 



Paired Differences 


df 

Sig. (2- 
tailed) 

Mean 

Std. 

Devi- 

ation 

Std. 

Error 

Mean 

95% Confidence 
Interval of the Dif- 
ference 

Lower 

Upper 

Pair Gain 1 Compter 
1 Gain2PPH 

1.179 

-.322 

.57 7 

-.006 

2.363 

2.041 

27 

.051 


Figure 2. Results on Web-based homework and paper-and-pencil homework 
for students who completed both. 

technical difficulties and were not able to complete the assignment. Another 
student in this group did not complete his paper-and-pencil homework. 
Consequently, we analyzed the remaining 28 students (52%) when comparing 
Web-based homework to paper-and-pencil homework. 

These percentages may be somewhat misleading because in Class D only 3 
out of 15 students who had Internet did the assignment at home as instructed, 
but 10 of the 15 students completed the assignment in the morning on com- 
puters at the school. Because those 10 students did not actually do the Web- 
based assignment at home, we chose to count those students as not having com- 
pleted the Web-based homework assignment. This class had a substitute teacher 
that day, and we speculate that the substitute may not have impressed upon 
the children the importance of doing the assignment at home, which may have 
accounted for the lower than expected participation rate in this class. When 
we exclude Class D from the above analysis, the participation rates increase, as 
shown in Table 3. 

For the following analyses, r-tests were run on the Web-based gain scores 
from pretest to posttest and on the paper-and-pencil gain scores from pretest 
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-3 -2 -1 0 1 2 3 4 5 6 


Gain Score 


Figure 3. Results on Web-based homework and paper-and-pencil homework 
for students who completed both. 

to posttest. There was learning in both conditions; however, when comparing 
the effect of Web-based homework and paper-and-pencil homework, including 
only those students who completed both the Web-based homework and the pa- 
per-and-pencil homework, there was a statistically reliable difference in favor of 
the Web-based homework condition. The paired 2-test, 2(27) = 2.0 4, p = 0.051, 
showed an effect size of .61 (see Figure 2). The 95% confidence interval for this 
effect size of .61 is (0.08-1.15). The mean gain for the Web-based homework 
group was 2.32 points out of 10 points, and for the paper-and-pencil home- 
work group the gain was 1.14 points out of 10 points. 

As shown in Figure 3, our analysis is not sensitive to one or two students. The 
weighted sum of gain scores, if we disregard negative scores, is 72 (Web-based 
homework) versus 36 (paper-and-pencil homework). 

Implications 

In this study, students learned significantly more with Web-based homework 
than with paper-and-pencil homework, and the effect size we reported of .61 
is large compared to other possible interventions shown in an in-depth study 
of the effect sizes of more than 100 classroom innovations accumulated from 
thousands of studies (Hattie, 1999). In the absence of one-to-one computing 
programs, teachers could use Web-based homework-assistance systems in their 
school computer labs or assign them for homework to students who have Inter- 
net access at home, access to school computers after school, or access to com- 
puters at the local library. The implications of this study could be important to 
policy makers, particularly considering the popularity of one-to-one computing 
initiatives (Hu, 2007) and claims that laptops can now be produced for prices 
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as low as $200 each (Bray, 2007). The cost of an intervention is of concern at a 
time when school budgets are already stretched thin, so policy makers need to 
see programs that can increase learning dramatically at low costs. In this study, 
we showed an intervention that led to a dramatic increase in student learning 
at a relatively low cost. For example, the Maine Learning Technology Initiative 
(2002-2004) was able to supply laptops to all of their seventh and eighth grade 
students for $300 per student per year, which is about one third of the cost of 
reducing class size. The cost of one-to-one computing programs could drop 
further, considering the new smaller and less expensive laptops being developed 
(Bray, 2007), and when excluding the growing number of students who already 
have computers at home. 

Caveats are in order, of course. Our study looked at the impact of learning 
at home, not in the classroom, so this intervention would help kids learn from 
their homework, not learn more effectively during class time. Other stud- 
ies have documented large gain scores when comparing traditional clsssroom 
instruction to computer-based tutoring (Kulik & Kulik, 1991; Razzaq, Men- 
dicino & Heffernan, 2008), so it is perhaps not a leap to assume the computer 
would also help learning when doing homework. Our study was also short 
term, and we were not able to do a retention test afterward due to a lack of 
time, as this study took place at the end of the school year. 

Our study was limited to fifth graders working on their math homework. We 
do not know if our results would generalize to students in other grades or other 
subjects. One limitation of the ASSISTment system is that it is not able to grade 
open responses or essay-type questions, and teachers are limited to multiple- 
choice or short-answer questions. Currently, this would limit us to tutoring sub- 
jects like math and science, and we do not know if our results would generalize 
to subjects such as English or history. 

In this study, we were also limited by the number of students who did not 
have Internet at home. As the digital divide narrows and more K— 12 students 
have access to computers and Internet at home, more teachers can take ad- 
vantage of the promise of Web-based homework-assistance systems. With this 
paper, we believe we have taken a step toward justifying the cost of providing 
students with the means to do their homework via the Web. 

CONCLUSION 

By using a system such as the ASSISTment system, students can learn more 
than they would by doing their homework with paper and pencil. Students get 
immediate feedback on their answers and help when they need it. In addition 
to better learning results, teachers can take advantage of the convenience of 
having homework automatically graded and recorded. Students can also benefit 
from Web-based homework because they may take their homework more seri- 
ously when they know it will be graded. 

With the ASSISTment system, teachers can also pinpoint exactly where 
students are having difficulties and get reports on which skills to address in class 
for individual students or the class as a whole (Feng, Heffernan & Koedinger, 
2006), thus allowing teachers to address shortcomings. Content is relatively 

Journal of Research on Technology in Education 343 

Copyright © 2009, ISTE (International Society for Technology in Education), 800.336.5191 
(U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. 



easy to develop in the ASSISTment system and can be created in a fraction of 
the time needed to develop content in other intelligent tutoring systems (Raz- 
zaq et ah, 2008). 

For future studies, we would like to determine if the results of this study have 
external validity. We would run the study with more students who have had 
more experience with the system. We would also like to find out if the results 
generalize for students in other grades and over longer periods of time. 
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APPENDIX A: MIXED PRETEST & NUMBER SENSE PRETEST 

Name Teacher 


1 . 

Turf Coverage 


Pounds 

Square Yards of Coverage 

6 

200 

10 

300 

14 

400 

18 

500 


The table above shows the number of pounds of turf needed to cover a given 
area. Based on the pattern in the table, how many pounds of turf are needed to 
cover 800 square yards? 

2. 2X + 2 = 14 

What value of X makes the equation shown above true? 
o A) X = 6 
o B) X = 8 

o C)X=10 
o D) X = 12 

3. The radius of a circle is 1 8 inches. What is the diameter of the circle? 

4. Mrs. Chipps wrote five numbers on the white board in her room. 

After class, one of the numbers was erased. The four numbers left are shown 
below. 


20 32 44 12 _L 

If the median of the five numbers that Mrs. Chipps wrote on the board was 20, 
which of the following could be true? 

o A) The number that was erased was greater than 44 
o B) The mode of the five numbers Mrs. Chipps wrote on the board was 
31 

o C) The number that was erased was less than or equal to 20 

o D) The mean of the five numbers Mrs. Chipps wrote on the board was 

56 

5. Mr. Lamb drew an equilateral triangle. Which of the following statements is 
true about the triangle? 
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o A) At least two angles are obtuse 
o B) At least one angle measures 90 degrees 
o C) All of the angles are less than 90 degrees 
o D) All of the angles have different measurements 

6. 


What is the area of the triangle shown above? 
o A) 220 cm 2 
o B) 156 cm 2 
o C) 125 cm 2 
o D) 225 cm 2 

7. 


Shelby created the input-output table shown above. Which of the following 
rules is true for all values of Shelby’s input-output table? 
o A) Input + 5 = output 
o B) Input times 5 = output 
o C) (Input times 4) + 2 
o D) (Input times 4) + 3 

8. Joe is 22 years older than Bob. If Joe is 43 years old now, how old is Bob? 

o A) 12 years old 

o B) 1 8 years old 

o C) 65 years old 

o D) 21 years old 

9. 

7 
(. 

6 
/ 

1 
3 
/ 
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On the coordinate grid above, which point is located at (7, 5)? 
o A 
o B 
o C 
o D 


10 . 


h 

S 

¥ 

3 

3 






r 















C, 
















A 



















D 

















bl 








1 


-I 









What are the coordinates of point C? 

Number Sense Pretest 


Name 


Teacher 


1. Which of the following is closest to the product of 397.8 * 10.3? 

o A) 3,000 

o B) 30,000 

o C) 400 

o D) 4,000 

2. Which of the following shows the numbers in order from least to greatest? 

o A) 0.452, 0.51, 0.432 
o B) 0.452, 0.432, 0.51 

o 0 0.432,0.51,0.452 
o D) 0.432, 0.452, 0.51 


3. 


«*+ 


+> 


-4 


-3 


-2 


-1 


Jeffrey is plotting points on the number line above. Between which two num- 
bers should Jeffery plot -3 14? 

4. What is 15/60 as a percent? 

5. Write 10/25 as a decimal. 


Journal of Research on Technology in Education 

Copyright © 2009, ISTE (International Society for Technology in Education), 800.336.5191 
(U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. 


349 


6. What is the value of the following expression? 


7*6 + 3 

7. Destinee has a total of 12 fish in her aquarium. Exactly 9 of the fish are gold- 
fish. What percent of the fish in the aquarium are goldfish? 
o A) 70% 
o B) 55% 
o C) 75% 
o D) 25% 



David made the circle shown below using gray and white triangles. What frac- 
tional part of the whole design is made up of gray triangles? Write your answer 
as a fraction. 

9. Kendra is going on vacation. She packed the following clothes in her suit- 
case. How many different outfit combinations will Kendra have to choose from 
during vacation? 


Suitcase for vacation 


Tops 

Pants 

T-shirt 

Sweatshirt 

Khaki pants 

Sweatpants 

Jeans 


H 1 1 — • — I 1 1 1 — I 1 — • — I H» 

40 80 120 160 


What is the distance between point C and point D on the number line shown 
below? 
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APPENDIX B: HOMEWORK PROBLEM SETS 
Homework/Mix Problems 

Name: Teacher: 

Date: 

1 . 


Fertilizer Coverage 


Pi Hinds 

Square Yards of Coverage 

4 

100 

8 

200 

12 

300 

16 

400 


The table above shows the number of pounds of fertilizer needed to cover a 
given area. Based on the pattern in the table, how many pounds of fertilizer are 
needed to cover 600 square yards? 

2. 2x + 2 = 10 

What value of x makes the equation shown above true? 
o A) x = 4 

o B) x = 6 

o C) x = 8 

o D) x = 12 

3. The radius of a circle is 14 inches. What is the diameter of the circle? 

4. Mr. Young wrote five numbers on the board in his classroom. After class, 
one of the numbers was erased. Four of the five numbers are shown below. 

18 25 30 17 ? 

If the median of the five numbers that Mr. Young wrote on the board was 18, 
which of the following could be true? 

o A) The number that was erased was greater than 30. 
o B) The mode of the five numbers Mr. Young wrote on the board was 
24. 

o C) The mean of the five numbers Mr. Young wrote on the board was 

22 . 6 . 

o D) The number that was erased was less than or equal to 18. 

5. Mr. Donato drew an equilateral triangle. Which of the following statements 
is true about the triangle? 

o A) At least one angle is obtuse, 
o B) All of the angles are acute, 
o C) At least one angle measures 90 degrees, 
o D) All of the angles have different measurements. 
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6. 



What is the area of the triangle shown above? 
o A) 126 cm 2 
o B) 210 cm 2 
o C) 252 cm 2 
o D) 420 cm 2 


7. 


Input 

1 

2 

3 

4 

5 

6 

7 

Output 

4 

6 

8 

10 

1 2 

14 

16 


Bridget created the input-output table shown above. Which of the following 
rules is true for all values in Bridget’s input-output table? 
o A) Input + 3 = Output 
o B) Input * 3 = Output 
o C) (Input * 2) + 1 = Output 

o D) (Input * 2) + 2 = Output 

8. Sam is 37 years older than Dennis. If Sam is 55 years old now, how old is 
Dennis? 

o A) 12 years old 

o B) 1 8 years old 

o C) 28 years old 

o D) 92 years old 


9. 



On the coordinate grid above, which point is plotted at (4, 3)? 
o A 
o B 
o C 
o D 
o E 
o F 
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What are the coordinates of Point A? 
o A) (6, 10) 
o B) (5, 9) 
o C) (9, 6) 
o D) (6, 9) 

Homework/Number Sense Problems 


Name: Teacher: 

Date: 

1. Which of the following is closest to the product 298.7 * 10.1? 

o A) 300 

o B) 2,000 

o C) 3,000 

o D) 20,000 

2. Which of the following shows the numbers in order from least to greatest? 

o A) 0.765, 0.82, 0.791 

o B) 0.765, 0.791, 0.82 

o C) 0.791, 0.82, 0.765 

o D) 0.791, 0.765, 0.82 

3. 

«H 1 1 1 1 1 h* 

- 3 - 2-1 0 1 2 3 


Marta is plotting points on the number line above. Between which two num- 


bers should Marta plot -214? 

o 

A) 1 and 2 

o 

B) 2 and 3 

o 

C) -2 and -1 

o 

D) -3 and -2 


4. Write 12/30 as a percent. 
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5. Write 15/25 as a decimal. 


6. What is the value of the following expression? 

3 + 6*4 

7. Judith has a total of 8 fish in her aquarium. Exactly 6 of the fish are guppies. 
What percent of the fish in the aquarium are guppies? 

o A) 48% 
o B) 60% 
o C) 68% 
o D) 75% 


8 . 



Key 

□ represents I gray tile 
] represents 1 white tile 


Shing made the design shown above using gray square tiles and white square 
tiles. What fractional part of the whole design is made up of gray tiles? Write 
your answer as a fraction. 


Salad Ingredients 


Lettuce 

Vegetable 

Dressing 

Iceberg 

Komaine 

Carrol 

Celery 

Broccoli 

Cauliflower 

Ranch 

Italian 

Caesar 

Vinaigrette 

French 


Rae is making a salad. The choices for the ingredients are shown in the chart 
above. What is the total number of different salads she can make using one let- 
tuce, one vegetable, and one dressing? 

10 . 

A B 

+4 1 — 1 - » -1 1 1 1 — I 1 — • — I (-► 

200 300 400 500 

What is the distance between point A and point B on the number line shown 
above? 
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APPENDIX C: SAMPLE COMPUTER PROBLEMS 

“2005_5_gr6” (Problem ID: 12330) [MA- 2003 - SPRING - 5] 
Which of the following is closer to the product 298.7 * 10.1? 

A) 300 

B) 2,000 

C) 3,000 

D) 20,000 

“2005_23_gr6” (Problem ID: 12395) [MA - 2005 - SPRING - 23] 
Which of the following shows the members from least to greatest? 

A) 0.765, 0.82, 0.791 

B) 0.765, 0.791. 0.82 

C) 0.791, 0.82, 0.765 

D) 0.791, 0.765, 0.82 

3. “2005_20_gr6” (Problem ID: 23332) [MA - 2005 - SPRING - 20] 

1 1 1 1 1 F* 

-3 - 2-1 0 1 2 3 


Marta is plotting points on the number line above. Between which two num- 
bers should Marta plot -2 Vii 
o A) 1 and 2 
o B) 2 and 3 
o C) -2 and -1 
o D) -3 and -2 

2. “2005_6_gr6” (Problem ID: 12341) [MA - 2005 - SPRING - 6] 

2x + 2= 10 


What value of x makes the equation shown above true? 
o A) x = 4 
o B) x = 6 
o C) x = 8 
o D) x = 12 


3. “2005_12_gr6” (Problem ID: 12361) [MA - 2005 - SPRING - 12] 

The radius of a circle is 14 inches. What is the diameter of the circle? 

4. “2005_15_gr6” (Problem ID: 12366) 

Mr. Young wrote five numbers on the board in his classroom. After class, one 
of the numbers was erased. Four of the five numbers are shown below. 

18 25 30 17 _? 
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If the median of the five numbers that Mr. Young wrote on the board was 18, 
which of the following could be true? 

o A) The number that was erased was greater than 30. 
o B) The mode of the five numbers Mr. Young wrote on the board was 
24. 

o C) The mean of the five numbers Mr. Young wrote on the board was 

22 . 6 . 

o D) The number that was erased was less than or equal to 18. 


APPENDIX D: SAMPLE HINTS 

3 .) 

1 1 1 1 1 h* 

- 3 - 2-1012 3 

Marta is plotting points on the number line above. Between which two num- 
bers should Marta plot -27 2 ? 


Answers: 

A) 1 and 2 

B) 2 and 3 

C) -2 and -1 

D) -3 and -2 

lefl side right side 

1 1 1 1 1 h* 

- 3 - 2-1 0 1 2 3 

-27, should be 27, units away from zero. What side of zero should -27 2 be on? 

Answers: 

the left side 
the right side 

Hint 1: 

The numbers to the right of zero on the number line are positive and are greater 
than zero. The numbers to the left of zero on the number line are negative and 
are less than zero. 

Hint 2: 

Marta should plot -27, on the left side of zero because it’s negative. Between 
which two numbers should Marta plot -21/2? 

Answers: (Interface Type: RADIO_BUTTON) 

A) 1 and 2 

B) 2 and 3 
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C) -2 and -1 

D) -3 and -2 


Hint 1: 

Marta should plot -2 '/ on the left side of zero, between two negative numbers. 
Hint 2: 

-27 2 should be 27, units away from zero. Is that to the left of -2 or to the right 
of -2? 

Hint 3: 

-27, should be plotted between -3 and -2. Choose D. 

4. Write 12/30 as a percent. 

Answers: 

40% 

40 

First, it will help to reduce the fraction to tenths. What is 12/30 reduced? 

Answers: 

3/10 
4/10 
4/5 
6/12 
Hint 1: 

What is a common factor between 12 and 30 that will give you 10 in the de- 
nominator? 

Hint 2: 

The common factor is 3. Divide 12 and 30 by 3. 

Hint 3: 

12/30 = 4/10 

Hint 4: 

Choose 4/10. 

Good. What is 4/10 in percent? 

Answers: 

0.4% 

0.04% 

4% 

40% 
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Hint 1: 

There are several ways to change a fraction to a percent. One way is to divide 
the numerator by the denominator and move the decimal point 2 places to the 
right. 

Hint 2: 

What is 4 divided by 10? 

Hint 3: 

40% 

4/10 = 0.4. Now move the decimal point 2 places to the right. 

Hint 4: 

0.4 = 40%. Choose 40%. 

5. Write 15/25 as a decimal. 

Answers: 

0.6 

First, it will help to reduce the fraction. What is 15/25 reduced? 

Answers: 

3/5 

Hint 1: 

What is the greatest common factor between 15 and 25? 

Hint 2: 

The greatest common factor is 5. Divide 15 and 25 by 5. 

Hint 3: 

15/25 = 3/5 

Hint 4: 

Type in 3/5 

Good. What is 3/5 in decimal? 

Answers: 

0.6 

Hint 1: 

One way to change a fraction to a decimal is to divide the numerator by the 
denominator, but it is easier to convert this fraction to tenths first. 

Hint 2: 

3/5 * 2/2 = 6/10. To divide by 10, move the decimal point one place to the left. 
(Since there is no decimal point, think of 6 as 6.0 and move the decimal point 
one place to the left.) 
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Hint 3: 

6/10 = 0.6 or 0.60 

Hint 4: 

Type in 0.6. 
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